Cautious Classifiers

The evaluation and use of classifiers is based on the idea that a classifier is defined as a complete function from instances to classes. Even when probabilistic classifiers are used, these are ultimately converted into categorical classifiers that must choose one class (with more or less confidence) from a set of classes. Evaluation metrics such as accuracy/error, global cost, precision, recall, f-score, specificity, sensitivity, effectiveness, macro-average, logloss, MSE or the Area Under the ROC Curve (AUC) are usually defined for “complete” classifiers. In this paper we pursue the usefulness and evaluation of “cautious” or “partial” classifiers. A cautious classifier adds an extra class “unknown” to the set of the original classes. This “unknown” class represents the cases where the prediction is uncertain or not reliable. Now, in a cost-insensitive context, accuracy and error will not be directly related but indirectly, through the coverage index. We develop new measures, efficacy and capacity, which find a compromise between reducing the number of misclassified data (error) and reducing the number of unclassified data (abstention). Inspired by ROC analysis we introduce several techniques to choose from a set of cautious classifiers. For probabilistic classifiers we define a discretisation method for converting them into cautious classifiers by using a “caution window”. We develop new response graphs to show the way in which different classifiers behave according to the size of the window and the class bias. In a cost-sensitive context, cost matrices and confusion matrices can be directly extended to account for this new class. Moreover, we extend ROC analysis and AUC evaluations to these classifiers, by considering the degree of abstention as an additional dimension.